The invention relates to electronic devices, and more particularly to digital cameras and white balancing and/or autoexposure methods and circuitry.
There has been a considerable growth in the sale and use of digital cameras in the last few years. Nearly 10M digital cameras were sold worldwide in 2000, and this number is expected to grow to 40M units by 2005. This growth is primarily driven by the consumer's desire to view and transfer images instantaneously.
FIG. 2 is a block diagram of the various image processing components, collectively referred to as image pipeline, in a typical digital still camera (DSC). Color filter array (CFA) interpolation, gamma correction, white balancing, color space conversion, and JPEG/MPEG compression/decompression constitute some of the key image pipeline algorithms. Note that the typical color CCD consists of a rectangular array of photosites with each photosite covered by a filter: either red, green, or blue. In the commonly-used Bayer pattern one-half of the photosites are green, and one-quarter are red, and one-quarter are blue. And the color conversion from RGB to YCbCr (luminance, chrominance blue, and chrominance red) used in JPEG/MPEG is defined by:
                    Y        =                ⁢                              0.299            ⁢                                                  ⁢            R                    +                      0.587            ⁢                                                  ⁢            G                    +                      0.114            ⁢                                                  ⁢            B                                                  Cb        =                ⁢                                            -              0.16875                        ⁢                                                  ⁢            R                    -                      0.33126            ⁢                                                  ⁢            G                    +                      0.5            ⁢                                                  ⁢            B                                                  Cr        =                ⁢                              0.5            ⁢                                                  ⁢            R                    -                      0.41859            ⁢                                                  ⁢            G                    -                      0.08131            ⁢                                                  ⁢            B                              so the inverse conversion is:
                    R        =                ⁢                  Y          +                      1.402            ⁢                                                  ⁢            Cr                                                  G        =                ⁢                  Y          -                      0.34413            ⁢                                                  ⁢            Cb                    -                      0.71414            ⁢                                                  ⁢            Cr                                                  B        =                ⁢                  Y          +                      1.772            ⁢                                                  ⁢            Cb                              where for 8-bit colors the R, G, and B will have integer values in the range 0–255 and the CbCr plane will be correspondingly discrete.
The color of a scene in an image appears different depending on the illumination under which the image is taken. This is due to the fact that different illumination sources, such as daylight, incandescent light, and fluorescent light, have different power spectral distributions. As a result, an object may appear to have a color different than its true color. The color most affected by this phenomena is white. Hence, the problem of making a white object to appear as white under different illuminants is called white balancing. This problem is normally solved by adjusting the gains of the three primary colors red, green, and blue of the CCD/CMOS sensor. The capability to do white balancing automatically without any user intervention is called automatic white balancing (AWB).
It is relatively easy to white balance a professional camera having manual white balance controls. This is done by zooming into a pure white object such that the object occupies the full frame. By pushing the white balance button, the color channel gains are adjusted. This way the camera balances its electronics so that the object specified as white is viewed and treated as white. Whenever the dominant light source changes, such as going from outdoors to indoors or from sunlight to shadow, this process needs to be repeated. On some cameras, manual white balancing is done by selecting a predefined set of light source settings such as “indoor”, “sunny”, “cloudy”, etc.
As far as automatic white balancing is concerned, both hardware and software solutions have been proposed. The hardware solution normally consists of using the average color of the scene to zero-out any color bias via an electronic circuitry. The main assumption made here is that the average color in the scene is neutral or gray (colorless). For scenes involving strong, dominant color objects where there is a bias in the average color, this assumption does not hold and the white balancing circuits cannot properly compensate. The use of programmable processors in digital cameras has made it possible to introduce software solutions to AWB by merely examining the raw image data captured by the CCD/CMOS sensor. The advantage of such solutions is the flexibility obtained in software upgrading and reusability
The most widely used technique for automatic white balancing is based on the “gray-world” assumption. This assumption states that the average of the colors in a scene is neutral or colorless. Let kr, kg, and kb be the scale factors or gains for the R, G, and B channels, respectively. These scale factors are determined as follows
                              k          r                =                ⁢                  max          ⁢                                    {                                                R                  avg                                ,                                  G                  avg                                ,                                  B                  avg                                            }                        /                          R              avg                                                                        k          g                =                ⁢                  max          ⁢                                    {                                                R                  avg                                ,                                  G                  avg                                ,                                  B                  avg                                            }                        /                          G              avg                                                                        k          b                =                ⁢                  max          ⁢                                    {                                                R                  avg                                ,                                  G                  avg                                ,                                  B                  avg                                            }                        /                          B              avg                                          where Ravg, Gavg, and Bavg denote the average red, green, and blue values, respectively, in the image.
Another commonly used technique is based on the “white-world” assumption. The white world assumption states that the red, green, and blue values of the brightest point in the image should be the same. Based on this assumption, the gains are adjusted as follows
                              k          r                =                ⁢                  max          ⁢                                    {                                                R                  max                                ,                                  G                  max                                ,                                  B                  max                                            }                        /                          R              max                                                                        k          g                =                ⁢                  max          ⁢                                    {                                                R                  max                                ,                                  G                  max                                ,                                  B                  max                                            }                        /                          G              max                                                                        k          b                =                ⁢                  max          ⁢                                    {                                                R                  max                                ,                                  G                  max                                ,                                  B                  max                                            }                        /                          B              max                                          where Rmax, Gmax, and Bmax denote the maximum red, green, and blue values, respectively, in the image.
Although the above assumptions lead to white balancing in many situations, they fail when the scene contains a relatively large background or a large object having the same color. For example, images having a wide blue sky or ocean background fall into this category.
Some researchers have thus developed solutions by making weaker assumptions about the scene. Hubel et al, U.S. Pat. No. 6,038,339 discloses a color by correlation scheme where the convex hull of a number of reference colors in chromaticity space under a specified illuminant are compared to the colors of the captured image. By correlating the captured image to the reference convex hulls for a set of illuminants, the most likely illuminant is identified. Other solutions have included the use of a neural network to map image chromaticities into primary color gains (Funt et al., Learning Color Constancy, IS&T Proc. Fourth Color Imagining Conf. pp.58–60 (1996)), and finding the near neutral objects for removing color casts (Cooper et al., Color Segmentation as an Aid to White Balancing for Digital Still Cameras, Proc. SPIE, vol. 4300, 99.16141 (2001)).
Color temperature is measured in Kelvins (K). It indicates the color of a black body radiator when heated to that temperature expressed in Kelvins. This color depends on the spectral radiation distribution of the black body at the specified temperature. On the Kelvin scale, the lower the color temperature, the redder the object appearance becomes, and the higher the color temperature, the bluer the object appearance becomes. For example, sunlight has a color temperature of about 5500 K, and a 100-watt incandescent light bulb has a color temperature of 2800 K. The human eye automatically adjusts to color temperature. In other words, the color of an object as seen by the eye is perceived to be the same when the object is viewed under different light sources. This is not the case when the object is seen through a digital camera. The color of the object appears differently depending on the light source under which the image is captured. As far as color appearance in a digital image is concerned, studies have shown that viewers prefer to see more saturated or stronger colors in the image. For example, viewers prefer to see skin tones healthier, grass greener, and sky bluer than they actually are.
Because the spectral distribution of a captured color consists of the product of the source (illuminant) spectral power distribution and the scene spectral reflectances, changing the source spectral distribution alters the product. Indeed, given a surface reflectance S(λ), an illuminant spectral distribution E(λ), and the three primary spectral sensitivities of the standard observer or sensor, r(λ), g(λ), and b(λ), then the red, green and blue components R, G, and B of a captured color can be written as:
                    R        =                ⁢                              k            r                    ⁢                      ∫                                          S                ⁡                                  (                  λ                  )                                            ⁢                              E                ⁡                                  (                  λ                  )                                            ⁢                              r                ⁡                                  (                  λ                  )                                            ⁢                              ⅆ                λ                                                                            G        =                ⁢                              k            g                    ⁢                      ∫                                          S                ⁡                                  (                  λ                  )                                            ⁢                              E                ⁡                                  (                  λ                  )                                            ⁢                              g                ⁡                                  (                  λ                  )                                            ⁢                              ⅆ                λ                                                                            B        =                ⁢                              k            b                    ⁢                      ∫                                          S                ⁡                                  (                  λ                  )                                            ⁢                              E                ⁡                                  (                  λ                  )                                            ⁢                              b                ⁡                                  (                  λ                  )                                            ⁢                              ⅆ                λ                                                        Of course, in a CCD with a color filter array (CFA), each photosite only captures one of the three R, G, or B, and the other two are obtained by interpolation from neighboring photosites.
Automatic exposure in a digital still camera (auto exposure in FIG. 2) typically assesses luminance levels and adjusts the exposure (time and/or aperture size). Another approach as in the Nikon F5 camera, a neural network is trained on 30,000 images to relate exposure value to parameter values of the following image parameters: brightness of the entire image, brightness of the focus area, contrast of the entire image, contrast of the lower and upper parts of the image, color temperature, position of focus, and distance from the object.